Gas visualization arrangements, devices, and methods

ABSTRACT

Gas visualization in an image depicting a scene, for an example embodiment comprises capturing a first IR image depicting the scene at a first time instance and a second IR image depicting the scene at a second time instance; performing image processing operations on image data derived from said first IR image and from said second IR image, to generate a collection of data representing the location of gas in one of the first or second IR images; and generating a third image by adjusting pixel values in an image depicting the scene, dependent on pixel values of said collection of data. According to various embodiments, there is further provided further processing of the collection of data, and/or gas detection, before generation of the third image with adjusted pixel values.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/670,311 filed Nov. 6, 2012 and entitled “GAS VISUALIZATIONARRANGEMENTS, DEVICES, AND METHODS,” which is hereby incorporated byreference in its entirety.

U.S. patent application Ser. No. 13/670,311, filed Nov. 6, 2012 claimsthe benefit of and priority to U.S. Provisional Patent Application No.61/639,749 filed Apr. 27, 2012 and to EP Patent Application No.11188119.9 filed Nov. 7, 2011, which are incorporated herein byreference in their entirety.

TECHNICAL FIELD

Generally, embodiments of the invention relate to the technical field ofgas visualization. More specifically, different embodiments of theapplication relates to gas visualization using an infrared (IR) imagingdevice.

BACKGROUND

Infrared (IR) imaging devices, such as IR cameras, can be used to findgas in various applications. For example, the IR camera manufacturerFLIR has a cooled gas camera that is used for finding many differentgases.

Detecting and visualizing gas using IR techniques can be difficult sinceIR imaging devices typically can detect and represent 65,000 thermallevels in a radiometric IR image but only have 255 colors to representthis data on the display. First, the gas detection in current prior arttends to be resource consuming with regard to processing power due tocomplicated operations required to detect gas present in an IR image.Secondly, the visualization of gas in an IR image also requires somekind of translation between the high resolution radiometric

IR image and a displayed IR image. It is possible to work with level andspan to visualize some smaller portion of these 65,000 levels onto the255 color scale, this is however quite time consuming and it can be hardto adjust to the current gas present in an imaged scene. Furthermore, ifthere are large differences in temperature between objects in the imagedscene, pixels having relatively small temperature differences will bevisualized as having the same color, or very similar colors. Thereby,the color difference between gas and surrounding pixels in the image maybe non-existent or very small, meaning that it is not possible, or veryhard, to discern the visualized gas in the image with the human eye.

Examples of related art are found in U.S. Pat. Nos. 5,656,813,7,649,174, 5,656,813, and 7,649,174.

While the prior art is directed to gas detection, it is deficientbecause the conventional methods require too much computational power,do not provide accurate gas detection, and/or do not provide sufficientvisualization of the detected gas.

Therefore, there is still a need for improvements in passive camerasystems in order to increase the detection capability in terms ofdistinguishing between an infrared absorbing gas cloud and backgroundelements as well as improved visualization in a computationallyefficient matter.

SUMMARY

Embodiments of methods, arrangements and devices described hereinprovide techniques for performing improved visualization of gas presentin an imaged or depicted scene. Furthermore, one or more embodimentsprovide improved and/or computationally inexpensive gas detection. Forexample, one or more embodiments may provide methods and apparatuses forimproved gas detection and/or visualization using an IR imaging device.One or more embodiments may provide certain advantages over prior arttechniques, such as to improve computationally efficient gas detectionand/or visualization, enable detection of a small concentration of gas,enable detection of gas leaks (e.g., small or large gas leaks), provideeasily interpretable visualization of gas, and/or enable animplementation that demands a relatively low computational effort.

In accordance with one or more embodiments, methods for gas detectionand visualization in infrared (IR) image data depicting a scene compriseperforming image processing operations on image data derived from afirst IR image depicting the scene at a first time instance and from asecond IR image depicting the scene at a second time instance, togenerate a collection of data representing the location of gas in one ofthe first or second IR image; detecting gas within the scene bydetecting gas representing pixels in the first or second IR image basedon said collection of data; and generating a gas visualizing image byadjusting, in an image depicting the scene, pixel values of pixelscorresponding to the gas representing pixels in one of said first orsecond IR image, such that the gas representing pixels aredistinguishable in the gas visualizing image.

In accordance with one or more embodiments, arrangements and devices forgas detection and visualization in infrared (IR) image data depicting ascene comprise devices and functionality for performing image processingoperations on image data derived from a first IR image depicting thescene at a first time instance and from a second IR image depicting thescene at a second time instance, to generate a collection of datarepresenting the location of gas in one of the first or second IR image;detecting gas within the scene by detecting gas representing pixels inthe first or second IR image based on said collection of data; andgenerating a gas visualizing image by adjusting, in an image depictingthe scene, pixel values of pixels corresponding to the gas representingpixels in one of said first or second IR image, such that the gasrepresenting pixels are distinguishable in the gas visualizing image.

For example, as in accordance with an embodiment, image processingmethods are disclosed to generate a collection of gas representing databy generating a temporal difference image. According to different methodembodiments, the difference image may be low-pass filtered and/ortransformed into the frequency domain. These and other embodimentmeasures enable gas visualization as moving or transient elements standout more clearly in a difference image, especially after low-passfiltering. Embodiments presented herein further enable gas detection byidentifying moving or transient elements. Gas visualization according toembodiments is achieved by generating a gas visualizing image byadjusting the pixel values of gas representing pixels in an imagedepicting the scene such that the gas representing pixels aredistinguishable with a high degree of processing efficiency.

In accordance with one or more embodiments, there is provided acomputing system configured to process infrared (IR) image data, thecomputing system comprising a memory configured to store infrared imagedata depicting a scene and a processor configured to process infraredimage data stored in the memory. The processor is further adapted toreceive from the memory a first IR image depicting the scene captured ata first time instance; receive from the memory a second IR imagedepicting the scene captured at a second time instance; perform imageprocessing operations on image data derived from the first and second IRimages to generate a collection of data representing the location of gasin one of the first or second IR images; and generate a third image byadjusting pixel values, in an image depicting the scene, dependent onpixel values of the collection of data.

Other embodiments of the claimed invention relate to computer-readablemediums, and computer program products on which are storednon-transitory information for performing gas visualization and/ordetection of gas present in an imaged or depicted scene.

The scope of the invention is defined by the claims, which areincorporated into this Summary by reference. A more completeunderstanding of embodiments of the invention will be afforded to thoseskilled in the art, as well as a realization of additional advantagesthereof, by a consideration of the following detailed description of oneor more embodiments. Reference will be made to the appended sheets ofdrawings that will first be described briefly.

BRIEF DESCRIPTION OF DRAWINGS

One or more embodiments of the present invention will be furtherexplained based on various embodiments and with reference to theaccompanying claims, in which:

FIG. 1 shows an example of an IR image of a scene with a gas leak.

FIG. 2A shows an example of a processed temporal difference image basedon a first and second IR image depicting the scene shown in FIG. 1, inaccordance with an embodiment of the invention.

FIG. 2B shows an example of a gas map, also referred to as a gas image,here in the form of a binary gas map, or binary gas image, in accordancewith an embodiment of the invention.

FIG. 2C shows an example of an intermediate gas image, in accordancewith an embodiment of the invention.

FIG. 2D shows an example of a gas visualization image, also referred toas a final gas image, in accordance with an embodiment of the invention.

FIG. 3 shows a block diagram of a method, in accordance with anembodiment of the invention.

FIG. 4A shows of a block diagram of the first part of a method, inaccordance with an embodiment of the invention.

FIG. 4B shows a continuation of the method in FIG. 4A, in accordancewith an embodiment of the invention.

FIG. 5 shows a block diagram of an edge detection method, in accordancewith an embodiment of the invention.

FIG. 6 shows a schematic view of an IR imaging system, in accordancewith an embodiment of the invention.

Embodiments of the invention and their advantages are best understood byreferring to the detailed description that follows. It should beappreciated that like reference numerals are used to identify likeelements illustrated in one or more of the figures.

DETAILED DESCRIPTION

Introduction

Embodiments of the claimed invention relate to methods, IR imagingdevices and/or IR imaging arrangements for performing gas detection andvisualization of gas present in an imaged, or depicted, scene.

According to an embodiment, an IR imaging device is realized as a devicewherein the units and functionality for performing, capturing, andprocessing of images are integrated in the device. An IR imaging devicemay for instance be realized as an IR camera. According to anembodiment, an IR imaging arrangement comprises one or more units andenables communication and/or transfer of data between the comprised oneor more units for performing, capturing, and processing of images.

Other embodiments of the claimed invention relate to computer-readablemediums on which are stored non-transitory information for performinggas detection and visualization of gas present in an imaged, ordepicted, scene.

The gas detection and visualization, for an embodiment, may be performedby identifying moving or transient elements, such as gas, in a capturedIR image frame depicting a scene, by producing a temporal differenceimage; process the difference image in order to enable detection of gas;and generating a gas visualizing image by adjusting the pixel values ofgas representing pixels in an image depicting the scene such that thegas representing pixels are distinguishable.

In FIG. 1, an example of an IR image 100 depicting a scene is shown,wherein there is a suspected gas leak. As generally known in the art, anIR image visualizes temperatures in a depicted scene. However, from thisrepresentation alone any gas present in the scene is typically notdistinguishable, as illustrated in FIG. 1.

FIGS. 2A to 2D show an example of images in different steps of gasdetection and visualization according to a method embodiment.

FIG. 2A shows an example of a processed temporal difference image 600representing the difference between two IR image frames captured usingthe same IR imaging system at two different time instances. As can beseen in the figure, a gas formed entity is shown without any relation tothe environment in the depicted scene.

FIG. 2B shows an example of a gas map 800, also referred to as a gasimage 800, wherein gas representing pixels in the IR image 100 of FIG. 1have been detected and have been assigned values that distinguish thegas representing pixels from the remaining pixels. In the image 800 ofFIG. 2B the gas representing pixels have all been assigned the value 1,or white, while the remaining pixels have all been assigned the value 0,or black, resulting in a binary gas map, or binary gas image. As will befurther explained below, this binary gas map or binary gas image is usedto improve processing efficiency.

FIG. 2C shows an example of an intermediate gas image 600, generated bycombining the processed difference image 600 of FIG. 2A with the gas map800, or gas image 800, of FIG. 2B.

FIG. 2D shows an image 1100 wherein the detected gas information isvisualized, obtained through a combination of the IR image 100 of FIG. 1and the intermediate gas image 800 of FIG. 2C. Such a combined image1100, wherein the depicted scene is visualized and any gas present inthe depicted scene is clearly visualized in relation to the depictedscene, will be presented in real time, or near real time, to the user oroperator of the gas detection and visualization method, IR imagingdevice and/or IR imaging arrangement according to the embodimentsdescribed below.

Gas Detection and Visualization Method

In FIG. 3, a block diagram schematically shows a general methodaccording to an embodiment of the invention. This embodiment method forgas visualization in an image depicting a scene comprises:

In step S310: capturing a first IR image depicting the scene at a firsttime instance.

According to an embodiment, the captured first IR image is depicting aspecified wavelength range within the infrared (IR) wavelength range.

In step S320: capturing a second IR image depicting the scene at asecond time instance.

According to an embodiment, the captured second IR image is depictingthe same wavelength range as the first IR image.

In different embodiment variants the captured first and second IR imagesmay be intermediately stored in a data memory for later processing ormay be transferred to and received in a data processor preferably in anIR imaging device or IR imaging arrangement in real time or close toreal time. An exemplifying application in which these IR images areintermediately stored is when IR images are captured and collected usingan IR imaging device at one time, whereas gas detection and gasvisualization is performed later using functions according toembodiments of the invention implemented in the IR imaging device or IRimaging arrangement, or implemented in a separate analyzing softwareoperable on a computer.

In step S330: performing image processing operations on image dataderived from the first IR image depicting the scene at a first timeinstance and from the second IR images depicting the scene at a secondtime instance to generate a collection of data representing the locationof gas in one of the first or second IR image.

According to an embodiment, the operations to generate a collection ofdata representing the location of gas comprise generating a temporaldifference image based on the first IR image and the second IR image.

According to an embodiment, generating a collection of data representingthe location of gas comprises further processing of the difference imagebefore generating said third image. According to an embodiment, furtherprocessing of the difference image comprises transforming the differenceimage into the frequency domain, e.g. by a fast Fourier transform (FFT)operation or by a power spectral density (PSD) operation. According toanother embodiment, processing of the difference image comprises lowpass filtering of the collection of data.

According to an embodiment, the method further comprises stabilizing thefirst and second IR images before the temporal difference image isgenerated, either by stabilizing the first IR image with respect to thesecond IR image, or by stabilizing the second image with respect to thefirst IR image.

According to an embodiment, the image processing operations comprisedetecting gas within the scene by detecting gas representing pixels inthe first or second IR image of the previous step based on or dependenton the collection of data. According to an embodiment, the first andsecond IR images are stabilized before gas detection is performed.According to an embodiment, detecting gas within the scene comprisesdetecting gas representing pixels in the first or second IR image basedon the difference image; and generating a third image or gasvisualization image by adjusting, in an image depicting the scene, pixelvalues of pixels corresponding to the gas representing pixels in one ofsaid first or second IR image, such that the gas representing pixels aredistinguishable.

In step S340: generating a third image by adjusting pixel values in animage depicting the scene, dependent on pixel values of said collectionof data.

According to an embodiment, generating a third image comprises adjustingpixel values in an image depicting the scene, dependent on a generateddifference image, for instance by adding the difference image to theimage depicting the scene. According to an embodiment, the differenceimage has previously been low-pass filtered in step S330.

According to an embodiment, the difference image or low-pass filtereddifference image may further have been adjusted according to apredetermined color or grey scale palette, as described further below,before it is added to the image depicting the scene. According to anembodiment, the difference image may be multiplied by a factor, e.g.between 0 and 1, before it is added to the image depicting the scene,thereby adding information according to an opacity determined by saidfactor.

According to an embodiment, generating a third image comprisesgenerating a gas visualizing image by adjusting in an image depictingthe scene pixel values of pixels corresponding to the gas representingpixels in the first or second IR images of the previous step such thatthe gas representing pixels are distinguishable in the gas visualizationimage.

Gas Visualization Image and Gas Location Representing Data

There are different options for generating the gas visualization image.According to different embodiments, the image in which the pixel valuesof pixels corresponding to the detected gas representing pixels areadjusted such that the gas representing pixels are distinguishable is aselected IR image depicting the scene, captured using an IR imagingdevice comprised in the IR imaging device. This may for example be thefirst IR image or the second IR image of the above steps. Alternatively,it may be another selected IR image depicting the scene at some othertime instance and preferably being captured by the same IR imagingsystem or IR detector as the mentioned first and second IR images. Inanother embodiment the gas visualizing image is generated based on avisual image, more specifically by adjusting, in a visual imagedepicting the scene, the pixel values of pixels corresponding to thedetected gas representing pixels, the visual image having apredetermined relation to the first and second IR images and beingcaptured using a visible light imaging system preferably comprised in anIR imaging device used to capture the mentioned first and second IRimages.

When generating the gas visualization image it is practical to base thegas visualization image on a selected image depicting the scene andadjust only the pixel values of pixels corresponding to the detected gasrepresenting pixels. Optionally, the gas visualization image may begenerated by adjusting also pixels corresponding to the non-gasrepresenting pixels based on image data of an image depicting the scenein order construct a suitably displayable image. This may for example bethe case when a fusion image comprising IR image data and visual imagedata with detected gas visualized is generated as a gas visualizationimage. An important thing is in any case that the location of the gas issufficiently accurately positioned in the depicted scene.

According to an embodiment, the collection of data representing thelocation of gas (i.e. gas location representing data) in the imagecomprises image pixel coordinates of image pixels representing gas.According to an embodiment, the collection of data has the form of a gasmap, or gas image, wherein pixels having coordinates that coincide withthe location of gas, in other words gas representing pixels, have pixelvalues that distinguish them from the remaining pixels. Differentembodiments relating to such gas maps, or gas images, are presentedbelow.

According to an embodiment, the image processing operations comprisecreating a temporal difference image based on or derived from the firstIR image and the second IR image, i.e. dependent on image data in thefirst IR image and the second IR image. According to an embodiment, gaspresent in the scene is made apparent by low-pass filtering thedifference image, thereby removing noisy areas that may otherwise beinterpreted as gas, for instance by a user viewing the difference imageor in an optional gas detection step 412, described further below.According to an embodiment in which a combined image is generated,removal of noise from the difference image further gives the effect ofavoiding that noise from being added to the combined image. The combinedimage is also referred to as third image or gas visualization imageherein.

In further detail according to an embodiment, generating a gasvisualizing image such that the gas representing pixels aredistinguishable comprises selecting in the temporal difference image,pixels representing gas based on said collection of gas locationrepresenting data.

Embodiments of Gas Detection and Visualization

FIG. 4A and 4B show embodiments of the inventive gas detection andvisualization method, wherein steps 401 to 412 relate mainly to gasdetection, while steps 414 to 418 relate to visualization of thedetected gas information.

In step 401 a an IR image of the scene is captured, at a first timeinstance. The capturing results in a first (1st) IR image 100. Anexample of such an IR image 100 is shown in FIG. 1.

In step 401 b an IR image of the scene is captured from the samedirection, i.e. with the same view, at a second time instance,preferably close in time to the first time instance and depicting thesame wavelength range as the first IR image. The capturing results in asecond (2nd) IR image 200.

The first and second IR images are captured using the same IR imagingsensor, the IR imaging sensor being comprised in the IR imaging device.

The second IR image is also referred to as the current IR image, orcurrent IR image frame, and is captured at the current time instance.The first IR image is also referred to as the previous IR image, orprevious IR image frame, and is captured at a previous time instance,previous to the first time instance. According to an embodiment, theprevious IR image and the current IR image are two subsequent IR imageframes in an image frame sequence, captured at time instance (t-1) and(t), respectively. According to this embodiment, the IR imaging devicemay be an IR video camera adapted to capture a sequence of IR imageframes.

The steps 402 to 412 relate to deriving image data from the first andsecond captured images and performing image processing operations on theimage data derived from the first and second images to generate acollection of data representing the location of gas in the image. Thecollection of data may for instance be represented as an image or a map.

In an optional step 402 either the first IR image is stabilized withrespect to the second IR image, or the second IR image is stabilizedwith respect to the first IR image, to compensate for movements of theIR imaging device etc. The stabilization is preferably performed beforethe temporal image is generated.

Different ways of accomplishing image stabilization are well known inthe art, and may be roughly divided into: optical image stabilizationmethods, wherein one or more physical optical element, e.g, lens,sensor, detector, is moved to compensate for the motion of the imagingdevice; and digital image stabilization, wherein the electronic image isshifted from frame to frame in order to compensate for the motion of theimaging device, based on detected movements of e.g. pixels or objectsidentified in the images. Image stabilization systems are commonly usedin visual imaging devices in order to compensate for movements of theimaging device.

Stabilization reasons and options are further explained below, inconnection with step ₄04.

In step 404, a temporal difference image 400 is generated, based on, orderived from, the first IR image and the second IR image. In otherwords, the first and second IR images are combined in such a way that atemporal difference image 400 is obtained.

The combination may for instance consist in subtracting the first imagefrom the second image, or to subtract the second image from the firstimage, thereby obtaining an image representing the difference betweenthe compared images. Since the difference image will compriseinformation on changes between the first and second IR image frames,moving elements will be visible in the difference image. Moving elementsin this context may for instance be transient elements, such as gas.

In order to obtain a good difference image, the photometric values oftwo images used for generating a difference image must be compatible.Since the first and second IR images, depicting a real world scene, bothrepresent the infrared spectrum of light, their photometric values arecompatible.

Furthermore, it is advantageous if the two images are aligned, orregistered, so that corresponding pixels coincide. Therefore, thecompared first and a second IR image are captured in close succession,and by the same IR imaging sensor. Preferably, the two images are twosuccessive image frames in an image frame sequence. The reason forcapturing the images in close succession to each other is that the realworld scene will not have changed much from the first image frame to thesecond and the second image thereby comprises substantially the samescene as the first image, with minor differences caused by movements ofobjects in the imaged scene, movements of the IR imaging device orrandom noise. As can be readily understood by a person skilled in theart, a first and a second image frames captured at time instances farapart may be used for the methods described herein and provide a goodresult as long as the real world scene will not have changed much fromthe first image frame to the second and the second image therebycomprises substantially the same scene as the first image, and as longas the IR imaging device has not moved to much in relation to the imagedscene. This may for instance be true for a monitoring situation whereinmonitoring is performed over time with a camera fixedly mounted orplaced on a stand, or tripod, in front of the imaged scene. Bysubtracting the first image from the second image, or the second imagefrom the first image, a difference image is obtained, comprising thedifferences caused by movements of objects in the imaged scene,movements of the imaging device and/or random noise.

In the difference image all differences caused by movements of objectsin the imaged scene will be comprised, meaning that even information onvery small concentrations of gas, such as very small gas leaks, will becomprised in the difference image.

High spatial frequency content, representing edges and contours of theobjects in the scene, may appear in a difference image unless the imagedscene is perfectly unchanged from the first time instance to the second,and the imaging device, and consequently also the imaging sensor, hasbeen kept perfectly still. The scene may for example have changed fromone frame to the next due to changes in light in the imaged scene ormovements of depicted objects. Also, in almost every case the imagingdevice and sensor will not have been kept perfectly still, meaning thatall stationary or non-moving parts of the imaged scene will not appearin the same location in the first and the second IR image. If theimaging device is handheld, it is evident that there will be movementscaused by the user of the imaging device. If the imaging device isstationary, for example on a stand, vibrations of the imaging device orthe surroundings may cause movements of the imaging sensor. Therefore,it may be advantageous to perform the optional image stabilization ofstep 402 before generating the temporal difference image 400.

Reducing Irrelevant Information by Edge Detection and Removal

In an optional step 406, edge detection is performed on the differenceimage, resulting in a data collection comprising edge locationinformation, i.e. information on where edges are located in thedifference image. The data collection comprising edge locationinformation may be in the form of an edge map 500, also referred to asan edge image 500.

The optional edge detection may be performed according to any methodknown in the art, for instance comprising search-based methods, such assearching for local directional maxima of the gradient magnitude, and/orzero-crossing based methods, such as searching for zero crossings in asecond-order derivative expression, usually the zero-crossings of theLaplacian or the zero-crossings of a non-linear differential expression.Edge detection according to the above methods may also comprise imagesmoothing as a preprocessing step, for instance low-pass filtering orGaussian smoothing, per se known in the art.

In FIG. 5, an embodiment of an edge detection method is shown, whereinedges are detected in an IR image 450. According to alternativeembodiments, the IR image 450 is the first IR image 100; the second IRimage 200, or another IR image captured using the same IR imaging deviceas has been used to capture the first and second IR images. According toanother embodiment, the IR image 450 is the difference image 400.However, using the difference image 400 introduces the risk that gasinformation is wrongfully detected as edges and therefore removed fromthe difference image.

In step S510, a high pass (HP) filter is applied to the IR image 450.Thereby, high frequency content in the IR image such as edge informationis detected, for example based on identification of local directionalmaxima of the gradient magnitude. From the HP filtering as HP filteredimage 515 is obtained. As described above, any suitable method may beused to obtain edge information from an image, HP filtering being onecommon example.

In an optional step S520, a threshold is applied to the HP filteredimage. From the optional threshold operation, a binary HP filtered imageis obtained wherein all pixels comprising the detected high frequencycontent are assigned a first pixel values while the remaining pixels areassigned a second value.

According to different embodiments, the threshold level is preset inproduction or calibration of the IR imaging device. Typically, thethreshold value depends on the general noise level in the imagescaptured by the IR imaging system of the IR imaging device or IR imagingarrangement, and the threshold value is set such that as few noiserepresenting pixels as possible will be wrongfully detected as gas,while the sensitivity for gas detection is as high as possible so thatno gas pixels are missed in the gas detection.

According to an embodiment, sensitivity for gas detection may beadjusted by an operator of the IR imaging device, using interactionfunctionality of the IR imaging device.

In the resulting HP filtered image 515, or the binary HP filtered imageif the optional step S520 has been performed, typically contains noise.In order to remove the noise, the HP filtered image 515, or the binaryHP filtered image, may be low pass (LP) filtered in optional step S530.

According to an embodiment, LP filtering is performed using an LPkernel, wherein the kernel values are adapted such that the LP filteringleads to expansion of the detected edges and removal of noise in theform of individual pixels, or a cluster of few pixels, comprised in theimage.

In general, a kernel filter works by applying a kernel matrix to everypixel in the image. The kernel contains multiplication factors to beapplied to the pixel and its neighbors and once all the values have beenmultiplied, the pixel value is replaced with for instance the sum of theproducts, or a mean value of the sum of the products. By choosingdifferent kernels, different types of filtering can be applied, as iswell known in the art.

For instance, the kernel may be a matrix of 3*3, 5*5, 7*7, 9*9 or anyother suitable size, and value of the kernel center pixel may be largerthan the value of the remaining kernel pixels. By means of example only,the kernel may have a center pixel value of 8 and remaining pixel valuesset to 1. Any other suitable values may however be used as well as anyselected weighting between different parts of the kernel meaning thatall pixels that are not the center pixel do not need to have the samepixel value.

According to an embodiment, the LP filter kernel is applied to eachpixel of the HP filtered image 515, or the binary HP filtered image,whereby the pixel value is set to the sum of the all of the pixel valuesof the HP filtered image 515, or the binary HP filtered image,multiplied/weighted by the values of the applied kernel. Alternatively,the pixel value is set to a mean value of obtained by dividing said sumwith the number of pixels in the applied kernel.

The different LP filtering embodiments all lead to the filtered pixelsbeing assigned a new pixel value taking into account the values of thesurrounding pixels in the original image. Thereby, a pixel thatoriginally has been set to a value indicating that it does not containedge information in the HP filtered image 515, or the binary HP filteredimage, may through the LP filtering be set to a value that indicatesthat it does contain edge information if a significant amount of thesurrounding pixels have values indicating that they contain edgeinformation. A significant amount of the surrounding pixels may forinstance be 50% of the surrounding pixels, two thirds of the surroundingpixels, or any other suitable fraction. This results in edge expansion,as continuous edge information is enhanced in the image when pixelssurrounding detected edges are included in the detected edges. Thisexpanding of the detected edges is thus performed before generating aprocessed difference image as explained below.

On the other hand, a pixel that originally has been set to a valueindicating that it contains edge information in the HP filtered image515, or the binary HP filtered image, may through the LP filtering beset to a value that indicates that it does not contain edge informationif a significant amount of the surrounding pixels have values indicatingthat they do not contain edge information. A significant amount of thesurrounding pixels may for instance be 50% of the surrounding pixels,two thirds of the surrounding pixels, or any other suitable fraction.This results in removal of noise in the form of individual or few pixelsthat are wrongfully detected as containing edge information since edgesdetectable in images consist of a number of subsequent pixels extendingin a defined direction.

Through the LP optional filtering of step S530, an LP filtered image 525is generated.

In step S540, a threshold is applied to the LP filtered image 525,resulting in the generation of an edge map 500, also referred to as anedge image 500.

According to an embodiment, pixels that have values above the thresholdvalue are identified as representing edge information while theremaining pixels are not identified as representing edge information.

According to an alternative embodiment, pixels that have values underthe threshold value are identified as representing edge informationwhile the remaining pixels are not identified as representing edgeinformation.

According to an embodiment, the edge map 500, or edge image 500, whereinthe pixels that have been identified as representing edge information,i.e. comprising the detected edge information, are assigned pixel valuesthat differ from the pixel values of the remaining pixels.Alternatively, the remaining pixels are assigned pixel values thatdiffer from the pixel values of the pixels that have been identified asrepresenting edge information, i.e. comprising the detected edgeinformation.

According to another embodiment, the pixels that have been identified asrepresenting edge information, i.e. comprising the detected edgeinformation, are assigned a first pixel value and the remaining pixelsare assigned a second pixel value. Optionally, the resulting edge map,or edge image, is a binary map, or binary image, wherein pixelsrepresenting detected edge information are assigned the pixel value 1and the remaining pixels are assigned the pixel value 0, or whereinpixels representing detected edge information are assigned the pixelvalue 0 and the remaining pixels are assigned the pixel value 1.

According to another embodiment, the pixels that have been identified asrepresenting edge information, i.e. comprising the detected edgeinformation, are assigned a pixel value, for instance 1 or 0, while thepixel values of the remaining pixels are left unchanged.

Alternatively, the remaining pixels are assigned a pixel value, forinstance 1 or 1, while the pixel values of pixels that have beenidentified as representing edge information, i.e. comprising thedetected edge information, are left unchanged.

Thereafter, in an optional step 408, a processed difference image isgenerated by combining the edge map, or edge image, 500 with thedifference image 400 such that the detected edge information is removed.

For instance, if the edge map, or edge image, is a binary map, or binaryimage, wherein pixels representing detected edge information have beenassigned the pixel value 0 and the remaining pixels have been assignedthe pixel value 1, then the edge map or binary image may be multipliedwith the difference image in order to obtain a processed differenceimage 600, wherein the edge information has been removed.

Alternatively, if the edge map, or edge image, is a binary map or binaryimage wherein the edge representing pixels have been assigned the pixelvalue 1 and the remaining pixels, i.e. pixels that have not beenidentified as representing edges, have been assigned the pixel value 0,then the gas map or binary image may be subtracted from the differenceimage, followed by a step wherein all pixels values <o are set to o, inorder to obtain a processed difference image 600, wherein the edgeinformation has been removed.

An example of such a processed difference image 600, wherein edges havebeen detected and removed, is shown in FIG. 2A.

As can be seen in FIG. 2A, only elements that differ between the firstand second IR image frames, such as transient elements, that furtherhave not been detected as edges remain in the processed differenceimage. In the processed difference image 600 of FIG. 2A, only transientelements in the form of a gas cloud remain, clearly showing the imagelocation of gas detected in the depicted scene.

As stated above, steps 406 and 408 are optional, but are shown tofurther improves the gas visualization performance since unwantedelements present in the difference image, i.e. elements that are notlikely to represent gas, are removed and therefore will not beemphasized by the visualization method in the final gas image obtainedin step 418 below. The edge detection and removal steps are especiallyadvantageous to include in images depicting scenes comprising straitlines, for instance outlines of buildings, walls, floors, roofs,streets, pipes, any rectangular or cylindrical objects or constructions,and so on. Therefore, it would typically be advantageous to include theedge detection and removal steps when the aim is to detect gas duringbuilding, construction, pipe, street or city inspections.

If the optional steps 406 and 408 have been performed, the processeddifference image is used as input to the transformation in the nextstep, step 410, and consequently the processed difference image is thebasis of the remaining steps presented in FIGS. 4A and 4B.

Further Processing of Difference Image

Steps 409 and 412 of FIG. 4A below are optional.

According to an embodiment, in Step 409 further processing of thedifference image, or processed difference image, is performed. In otherwords, the operations to generate a collection of data representing thelocation of gas comprises further processing of the difference imagebefore generating the resulting gas visualization image, also referredto as the third image in connection with FIG. 3. According toembodiments presented below, the further processing of step 409 maycomprise transforming the collection of data into the frequency domainin an optional step 410 by: an FFT operation; or a PSD operation.Alternatively, according to embodiments presented below, the furtherprocessing of step 409 may comprise low pass filtering the collection ofdata in an optional step 411. According to an embodiment, a combinationof steps 410 and 411 may be performed.

According to an embodiment, low pass (LP) filtering of the collection ofdata, in other words difference image 400 or processed difference image600, is performed in step 411, resulting in a low pass (LP) image 701.By low pass filtering the difference image 400 or processed differenceimage 600, noise present in the image 400, 600 is removed or reduced.Thereby any gas present in the scene becomes more clearlydistinguishable as unnecessary and disturbing information, in the formof noise, is no longer present in the image.

According to an embodiment, step 410 is performed. In the transformingstep 410, the difference image 400, or the processed difference image600 if the optional steps 406 and 408 have been performed, istransformed from the time domain into the frequency domain. Gasdetection is then performed in the transformed difference image toidentify pixel coordinates of pixels representing gas present in thescene. Hereinafter, the phrase difference image will be used in thetext, referring to either one of the difference image 400, or theprocessed difference image 600.

The transformation is performed by an operation according to apredetermined transformation function, typically a Fourier transformsuch as a fast Fourier transform (FFT, FFT2), or a transform based on aFourier transform, such as a discrete cosine transform (DCT) or a powerspectral density (PSD), which describes how the power of a signal ortime series is distributed with frequency. The inventor has found that aFourier transformation of the image into the frequency domain makes thefrequencies of the gas present in the image appear surprisingly well.

According to an embodiment, a PSD estimate is calculated using a per seknown Welch method described in Stoica et al., “Introduction to spectralanalysis” (Prentice Hall, 1997), wherein data segments are allowed tooverlap and can be represented as

$\begin{matrix}{{{x_{i}(n)} = {x\left( {n + {iD}} \right)}}\mspace{20mu} \left\{ \begin{matrix}{{n - 0},1,\ldots \mspace{14mu},{M - 1}} \\{{i = 0},1,\ldots \mspace{14mu},{L - 1}}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$

where the starting point for segment i is iD and the segment length isM. According to an embodiment M>D, meaning that there will be overlap,e.g. M=D2 corresponds to an overlap of 50%. The total length of thesignal is LD.

According to an embodiment, the i^(th) periodogram is calculated as

$\begin{matrix}{{{{\hat{X}}^{(i)}(f)} = {\frac{1}{MU}{{\sum\limits_{n = 0}^{M - 1}\; x_{{i{(n)}}{w{(n)}}\varepsilon^{{- j}\; 2\; \pi \; {fn}}}}}^{2}}}{{i = 0},1,\ldots \mspace{14mu},{L - 1}}} & \left( {{Eq}.\mspace{14mu} 2} \right)\end{matrix}$

Where U is a normalization factor that corresponds to the power in thewindow function, given by:

$\begin{matrix}{U - {\frac{1}{M}{\sum\limits_{n = 0}^{N - 1}\; {w^{2}(n)}}}} & \left( {{Eq}.\mspace{14mu} 3} \right)\end{matrix}$

The Welch power spectrum {circumflex over (X)}(f) is then defined as theaverage of the periodograms in Eq. 2, i.e.,

$\begin{matrix}{{\hat{X}(f)} = {\frac{1}{L}{\sum\limits_{t - u}^{L - 1}\; {X^{(i)}(f)}}}} & \left( {{Eq}.\mspace{14mu} 4} \right)\end{matrix}$

According to an embodiment, a Hamming window function defined as

$\begin{matrix}{{{w(n)} = {0.54 - {0.46\cos \frac{2\pi \; n}{M - 1}}}}{0 \leq n \leq \leq {M - 1}}} & \left( {{Eq}.\mspace{14mu} 5} \right)\end{matrix}$

is used together with a 50% overlap for the PSD estimates. As is readilyapparent to a person skilled in the art, different window functions anddifferent amounts of overlap may be applied according to circumstances.

According to an embodiment, the transformation of the difference imageinto the frequency domain is performed block wise for difference imageblocks of a predetermined size, using image blocks of a size smallerthan the size of the difference image. By means of example, the blocksize may be 2*2, 4*4, 8*8, 16*16, 32*32, or 32*24 pixels. Any suitableblock size may be chosen depending on circumstances, e.g. dependent onthe size of the difference image onto which the transform is applied.Generally it is advantageous to use blocks that are not too small inorder for low frequency information to be included and detectable withinthe block.

According to an embodiment, the block may be converted into an array,wherein the rows of the matrix are placed one after another. Thereafter,the resulting signal, in array format, is used for the subsequentfrequency transformation.

Optionally, the block wise transformation into the frequency domain isnot performed for every pixel in the difference image, but instead isperformed for a sub portion of the pixels of the difference image.Thereby, the transformation may also result in a down sampling of thedifference image, resulting in a down sampled frequency domain image700, i.e. frequency domain representation of the difference image. Forexample, a frequency transformation may be performed for every tenthpixel, meaning that the resulting frequency domain image 700 is downsampled ten times.

If a down sampling has taken place during the frequency transformationstep the signal, i.e. the frequency domain image 700, is up-sampled toits original size again after the frequency transformation. According todifferent embodiments the up-sampling may be performed using any kind ofinterpolation method per se known hi the art, such as for instancenearest neighbor interpolation, linear interpolation, bilinearinterpolation, polynomial interpolation, cubic interpolation, splineinterpolation, or cubic spline interpolation.

As is readily apparent to a person skilled in the art, any down samplingfactor may be selected dependent on circumstances. If the main aim is tomaintain as much information as possible in the transformation step, theblock wise transformation may be performed for every pixel, every otherpixel or every third pixel, for example. If on the other hand the aim isto reduce information in order to obtain computational efficiency whenperforming further calculations on the frequency domain image 700, alarger sampling distance may be selected, for instance resulting infrequency transformation of every tenth, twentieth or thirtieth pixel.Evidently, the selection of down sampling factor also depends on thesize of the difference image, i.e. how much information the differenceimage contains before the transformation. As is readily apparent to aperson skilled in the art a pixel wise transformation is possible,wherein no down- and up-sampling is necessary, but it will be morecomputationally expensive.

For a specified IR imaging device having specified IR optics, the sizeof the IR images, and thereby the size of the difference image, aretypically known. Therefore, an appropriate down sampling factor may bepreset during production or calibration of the imaging device, orpreselected by a user. According to an embodiment, the down samplingfactor is selected by the user and manually input into the IR imagingdevice during use.

According to an embodiment, each pixel is assigned the value of aselection of the largest frequency peak, the largest peak within the lowfrequency content or a peak in the low frequency content within thecorresponding transformed image block related to the pixel. According toanother embodiment, each pixel is assigned the value of the added pixelvalue of two or more such peaks.

Gas Detection in Transformed Difference Image

Step 412 is optional and comprises further processing of the differenceimage 400, or a processed version of the difference image in the form ofa processed difference image 600, frequency domain image 700 or LP image701. Below, the term difference image may refer to any of the images400, 600, 700 or 701.

According to an embodiment, step 412 comprises detecting gas within thescene by detecting gas representing pixels in the first IR image 100 orsecond IR image 200 based on the difference image 400, 600, 700, 701;and generating a third image, or final gas image 1100, by adjusting, inan image depicting the scene, pixel values of pixels corresponding tothe gas representing pixels in one of said first or second IR image 100,200, such that the gas representing pixels are distinguishable.

According to an embodiment, in step 412, gas detection is performed inthe transformed difference image, also referred to as the frequencydomain image 700. According to another embodiment, in step 412, gasdetection is performed in the LP image 701.

According to an embodiment, the gas detection is performed in thedifference image 400, 600, 700, 701 in order to identify the location ofpixels representing transient elements, such as gas, present in thedepicted scene. The location of pixels may typically be represented asthe coordinates of the pixels in the difference image 400, 600, 700,701.

According to an embodiment, the location of pixels representing gaspresent in the depicted scene is detected by LP filtering orthresholding the difference image 400, 600, 700, 701 according to a LPfiltering or threshold value, separating the gas representing imagepixels from the non-gas representing image pixels based on thedifference image 400, 600, 700, 701 pixel values.

As part of the gas detection step, low-pass filtering or thresholdingthe pixels of the difference image 400, 600, 700, 701 may in oneembodiment be used to generate a collection of data representing thelocation of gas in one of the IR images to create or generate a gas map,or a gas image 800 wherein the pixels that have been identified asrepresenting gas information, i.e. comprising the detected gasinformation, are assigned pixel values that differ from the pixel valuesof the remaining pixels. Alternatively, the remaining pixels areassigned pixel values that differ from the pixel values of the pixelsthat have been identified as representing gas information, i.e.comprising the detected gas information.

According to another embodiment, the pixels that have been identified asrepresenting gas information, i.e. comprising the detected gasinformation, are assigned a first pixel value and the remaining pixelsare assigned a second pixel value. Optionally, the resulting gas map, orgas image, is a binary map or image wherein pixels representing detectedgas information are assigned the pixel value 1 and the remaining pixelsare assigned the pixel value 0, or wherein pixels representing detectedgas information are assigned the pixel value 0 and the remaining pixelsare assigned the pixel value 1.

According to another embodiment, the pixels that have been identified asrepresenting gas information, i.e. comprising the detected gasinformation, are assigned a pixel value, for instance 1 or 0, while thepixel values of the remaining pixels are left unchanged. Alternatively,the remaining pixels are assigned a pixel value, for instance 1 or 1,while the pixel values of pixels that have been identified asrepresenting gas information, i.e. comprising the detected gasinformation, are left unchanged.

Gas Detection and/or Visualization Modes

According to an embodiment, combination of embodiments according to thefurther processing of steps 410, 411 and 412 represent different modesavailable in an IR imaging device or IR imaging arrangement. In otherwords, there are more than one mode for further processing, comprising aselection of transforming the collection of data into the frequencydomain by use of an FFT operation or a PSD operation; low pass filteringthe collection of data; and/or performing gas detection.

According to an embodiment a user of an IR imaging device or IR imagingarrangement is enabled to select a mode, or switch between modes, usingfor instance interaction functionality 4 described below in connectionwith FIG. 6. According to an embodiment, selectable modes are presentedto a user as menu options in a graphical user interface on a displayintegrated in or coupled to the IR imaging device. According to anembodiment, a mode is automatically selected in production, calibrationor during use, dependent on circumstances.

According to a use case embodiment, it may be beneficial to use a modecomprising transforming the collection of data into the frequency domainby use of an FFT operation or a PSD operation and gas detectionaccording to methods described above for an IR imaging device that ismore or less stationary. This may for instance be true for a monitoringtype camera that is fixedly mounted, or for a camera placed on a stand,or tripod. However, this mode may of course be used also for a handheldIR imaging device.

According to another use case embodiment, it may be beneficial to use amode comprising low pass filtering, and optionally also gas detectioncomprising thresholding, for a handheld camera that is likely to moveover time, relative to the imaged scene. The gas detection and/orvisualization will in this case often not be as exact as in the modedescribed above. However, this mode still provides a greatly enhancedimage with regard to visualization of gas, as compared to an imagewherein no further processing has been performed. However, this mode mayof course also be used in a monitoring situation wherein the IR imagingdevice is fixedly mounted or placed on a stand, or tripod.

According to an embodiment, if no specific mode has been selected, by auser or automatically dependent on circumstances, the display integratedin or coupled to the IR imaging device displays a regular IR, visuallight or combined image that has not been processed according toembodiments described herein.

Generating Gas Visualization Image

Steps 414 to 418 relate to generating a gas visualizing image byadjusting the pixel values of gas representing pixels in an imagedepicting the scene, such that the gas representing pixels aredistinguishable.

Step 414 is optional and may be performed if step 412 has beenperformed.

According to an embodiment, in step 414, an intermediate gas image 900is generated by combining the gas map, or gas image, 800 with thedifference image such that pixels corresponding to the pixels that havebeen identified as representing gas information, i.e, having beenassigned one or more gas identifying values in the gas map, or gasimage, are assigned the pixel values of the corresponding pixels in thedifference image. The pixel values of the remaining pixels in theintermediate gas image 900 are for instance set to 0, 1 or any othervalue selected to clearly separate or distinguish the remaining pixelsfrom the pixels corresponding to the pixels that have been identified asrepresenting gas information. This results in the intermedi ate gasimage 900 being a processed version of the difference image, wherein allpixel information from the difference image regarding the pixels thathave been identified as representing gas information is maintained,while all pixel information from the difference image regarding theremaining pixels is removed, e.g. by assigning all remaining pixels to asingle pixel value that is not dependent on the difference image.

For instance, if the gas map, or gas image, is a binary map or imagewherein the gas representing pixels have been assigned the pixel value 1and the remaining pixels, i.e. pixels that have not been identified asrepresenting gas, have been assigned the pixel value 0, then the gas mapor image may be multiplied with the difference image in order to obtaina processed difference image, wherein the non-gas pixel information isremoved.

Alternatively, if the gas map, or gas image, is a binary map or imagewherein the gas representing pixels have been assigned the pixel value 0and the remaining pixels, i.e. pixels that have not been identified asrepresenting gas, are assigned the pixel value 1, then the gas map orimage may be subtracted from the difference image, followed by a stepwherein all pixels values <0 are set to 0.

An example of an intermediate gas image 900, wherein pixels representinggas have been detected and assigned the pixel values of thecorresponding pixels in the difference image, and further wherein pixelinformation for all non-gas pixels has been removed, is shown in FIG.2C.

In an optional step 416, the pixel values of the image input from thepreviously performed step are adjusted.

According to an embodiment, the input image is a generated differenceimage 400, 600 or LP image 701, and adjusting the pixel values comprisesadjustment according to a predetermined color or grey scale palette, asdescribed further below.

According to an embodiment, an intermediate gas image with adjustedpixel values 1000 is generated. According to an embodiment, the pixelvalues of the gas representing pixels in the intermediate gas image 900are adjusted according to a predetermined palette of pixel values, togenerate an intermediate gas image with adjusted pixel values 1000.

The predetermined palette may comprise any appropriate selection ofrepresentations distinguishable from each other to the human eye, forinstance grey scale values, different intensity values, differentpatterns such as halftone patterns, different shades of a certain colorsuch as red, green or blue, or a scale comprising two three, or moredifferent colors of different hue, saturation or intensities. As isreadily apparent to a person skilled in the art, the palette maycomprise a single pixel value, such as a single color value with novariation in hue, saturation or lightness/intensity, resulting in a flatcolor representation when applied to the pixel values of the gasrepresenting pixels in the intermediate gas image 900.

Regardless of which representations are selected for the palette, therepresentations in the selected palette are mapped to the pixel valuesof the intermediate gas image 900, such that a certain pixel will berepresented by a certain pixel value according to the selected palette.According to an embodiment the mapping relationships are predetermined,for example during production or calibration the IR imaging device, orduring the development or implementation of the gas detection andvisualization method.

After adjustment of the pixel values in an intermediate gas image goo,an intermediate gas image with adjusted pixel values 1000 is thusobtained.

In step 418 a gas visualization image, also referred to as a final gasimage 1100, is generated.

According to an embodiment, the final gas image 1100 is generated byadjusting the pixel values of gas representing pixels in an imagedepicting the scene 1200, such that the gas representing pixels aredistinguishable.

According to an embodiment, the final gas image 1100 is obtained byadjusting pixel values of an LP image 701 according to a predeterminedpalette of pixel values, wherein the predetermined palette comprises aselection of the palette options presented above.

According to an embodiment, generating a third image comprises combininga generated difference image 400, 600 or LP image 701 with an image 1200depicting the scene. According to an embodiment, the combinationcomprises adjusting pixel values in an image 1200 depicting the scene,dependent on a generated difference image 400, 600 or LP image 701, forinstance by adding the difference or LP image to, or multiplying thedifference or LP image with, the image depicting the scene.

As described above in connection with step 416, the difference image400, 600 or LP image 701 may according to embodiments further have beenadjusted according to a predetermined color or grey scale palette beforeit is added to the image depicting the scene. According to anembodiment, the difference image 400, 600 or LP image 701 may bemultiplied by a factor, e.g. between 0 and 1, before it is added to, ormultiplied with, the image depicting the scene, thereby addinginformation according to an opacity determined by said factor.

According to an embodiment, the pixel values of the LP image 701 havebeen adjusted in step 416 in order to obtain an intermediate gas imagewith adjusted pixel values 1000, and wherein said intermediate gas imagewith adjusted pixel values 1000 is combined with an image depicting ascene in step 418.

According to an embodiment, gas detection has been performed in an LPimage 701 in step 412 thereby generating a gas map 800, where after thegenerated gas map may be used as input into further method steps,comprising any or all of the optional steps 414 to 418, according toembodiments described above.

According to an embodiment, the final gas image 1100 is obtained bycombining a selected image depicting the scene 1200 with theintermediate gas image 900 or the adjusted intermediate gas image 1000in such a way that the pixel values of the pixels in the image depictingthe scene 1200 corresponding to the gas representing pixels are adjustedbased on the pixel values of the pixels in the intermediate gas image900 corresponding to the gas representing pixels. Thereby a resultingfinal gas image 1100 is generated, wherein the depicted scene and gaspresent in the depicted scene is visualized, and wherein the locationrelation between the scene and gas present in the scene is substantiallyaccurate. In other words, the intermediate gas image goo is combinedwith an image depicting the scene 1200, to generate a final gas image1100 wherein gas present in the depicted scene is visualized.Alternatively, if the optional step 416 has been performed, the adjustedintermediate gas image 1000 is combined with an image depicting thescene 1200, to generate a final gas image 1100 wherein gas present inthe depicted scene is visualized.

There are several further different options and embodiments in thegeneration of the final gas image 1000. The final gas image may forexample be based on:

-   -   The adjusted intermediate gas image 1000 according to different        palettes;    -   The intermediate image only comprising the difference image;        and/or    -   Generating a cloud structure substantially covering the gas        pixels.

A cloud structure may be fitted onto the gas pixels or moreapproximately covering the gas pixels, for example oriented around thecenter of detected gas pixels. Such a cloud structure may be apredefined item such as a clip-art cloud stored in memory, and may besuperimposed on the final gas image. The cloud structure may be scalableor have a standardized size.

Options for Basic Image for Gas Visualization Image

According to embodiments, the image 1200 depicting the scene and beingused as a basis for the gas visualization image can be selected fromvarious options. In one embodiment the image 1200 is an IR image and thefinal gas image 1100 is generated by adjusting the pixel values of gasrepresenting pixels in the IR image 1200, wherein the IR image 1200 iseither the first IR image 100, the second IR image 200, or a third,different, IR image, depicting the scene, captured using the same IRimaging system that was used for capture of the first and second IRimages, the IR imaging system being comprised in the IR imaging device,such as an IR camera or IR imaging arrangement.

According to another embodiment, the image 1200, depicting the scene, isgenerated from a temporal difference image 400 or the processeddifference image 600, similarly adjusting the pixel values of pixelscorresponding to the detected gas representing pixels.

According to other embodiments, the final gas image 1100 is generated byadjusting the pixel values of gas representing pixels in an image 1200,depicting the scene, wherein the image 1200 has been captured using adifferent IR imaging system, a visible light imaging system, or anyother kind of imaging system adapted to receiving a certain range ofwavelengths of light and generating a visible representation of saidreceived light, comprised in or communicatively coupled to the IRimaging device, such as an IR camera, or IR imaging arrangement.

According to another embodiment, the image 1200, depicting the scene, isa fused image, the fused image being a result of a fusion between an IRimage depicting the scene and a visible light image, also referred to asa visual image, depicting the same scene. The fused IR image and visiblelight image used have been captured using an IR imaging system and avisible light imaging system, respectively, the imaging systems forinstance being comprised in the IR imaging device, such as an IR camera,or IR imaging arrangement or communicatively coupled to the IR imagingdevice, such as an IR camera, or IR imaging arrangement. According tothis embodiment, the final gas image 1100 is generated by adjusting thepixel values of gas representing pixels in the fused image. Fusion of anIR image and a visual light image may be performed in any method per seknown in the art and is not further described herein.

An example of a final gas image 1100 is shown in FIG. 2D. As shown inFIG. 2D, gas present in the scene is distinguishably visualized in thefinal gas image 1100, thereby making it easy for a person viewing thefinal gas image 1100 to interpret the image and visually detect whetherthere is gas present in the scene and, if that is the case, how much gasthere is and where in the scene the gas is located. In other words, aneasily interpretable visualization of gas is provided.

Gas Visualization Options

According to an embodiment, each pixel comprising detected gas isrepresented as 3 bits of information, wherein one bit of informationcomprises color information. The color information may for instance spanfrom different saturation levels of green for negative values todifferent saturation levels of red for positive values. For instance,the color values may have a range of −1 to 1, wherein −1 is equal tofully saturated green, 0 is equal to no color/no saturation, and 1 isequal to fully saturated red. According to this embodiment, the secondand third bits of information in each pixel comprise the pixel values ofthe original grey scale image. In other words, the pixel values of theoriginal image are not over-written, but kept in their original form,simply adding one bit of color information. In this way, the “gas cloudlook” is kept, since the gas is visualized in a transparent manner, theadded color information having varying opacity. This further improvesthe interpretability of the gas information visualized in the image,since the partially transparent cloud, or plume, visualized in the imageis very intuitively interpreted as gas.

As is readily apparent to a person skilled in the art, any suitablenumerical or other scale may be used to describe the range of the bitinformation, and any selection of color values, such as hue andsaturation, light/intensity and/or grey scale values may be used tovisualize the bit information.

According to another embodiment each pixel is represented as a three bitcolor using any known color space representation, for instance RGB,CIEXYZ or CIELab.

Alignment

If an IR image captured using the same IR imaging system as is used forcapturing the first and second IR images is used as a basis for thefinal gas image 1100, there is naturally no parallax between differentimaging systems. Thereby, there will be no parallax related pixeldisplacements that need to be compensated for when combining the imageinformation in the intermediate gas image with adjusted pixel valueswoo, or the intermediate gas image 900, with the IR image used as abasis for the final gas image 1100.

According to the embodiments wherein different imaging systems, such asfor instance two different IR imaging systems or an IR imaging systemand a visual imaging system, are used to capture image information, theoptical axes between the imaging components may be at a distance fromeach other and an optical phenomenon known as parallax will arise,causing parallax related pixel displacement between the images capturedwith different imaging systems. To eliminate the parallax relatederrors, arising from the parallax distance between the imaging systemsand an angle between the optical axes of the imaging systems, the imagesthat are to be combined in some way must first be aligned.

Therefore, after capture, the visual image and the IR image may bealigned to compensate for the parallax between the optical axes thatgenerally arises due to differences in placement of the sensors forcapturing said images and the angle created between these axes becauseof mechanical tolerances that generally prevents them being mountedexactly in parallel.

Computer Readable Medium

According to an embodiment of the invention, there is provided acomputer-readable medium on which is stored non-transitory informationfor performing a method comprising capturing a first IR image depictingthe scene at a first time instance; capturing a second IR imagedepicting the scene at a second time instance; performing imageprocessing operations on image data derived from the first and second IRimages to generate a collection of data representing the location of gasin one of the first or second IR image; detecting gas within the sceneby detecting gas representing pixels in the first or second IR image ofthe previous step based on the collection of data; and generating a gasvisualizing image by adjusting, in an image depicting the scene, pixelvalues of pixels corresponding to the gas representing pixels in thefirst or second IR image of the previous step, such that the gasrepresenting pixels are distinguishable in the gas visualizing image.

According to further embodiments, there is provided computer-readablemediums on which is stored non-transitory information for performing anyof the method embodiments described above.

System Architecture

FIG. 6 shows a schematic view of an embodiment of an IR imaging deviceor IR imaging arrangement 1 that comprises an IR imaging system 12having an IR sensor 20.

According to an embodiment, an IR imaging device is realized as a devicewherein the units and functionality for performing, capturing, andprocessing of images are integrated in the device. An IR imaging devicemay for instance be realized as an IR camera. According to anembodiment, an IR imaging arrangement comprises one or more units andenables communication and/or transfer of data between the comprised oneor more units for performing, capturing, and processing of images.Hereinafter, in the description of the system architecture, the terms IRimaging device and IR imaging arrangement will be used interchangeably,wherein the only difference lies in whether the units and functionalityare comprised in a single physical unit or device, or whether the unitsand functionality are communicatively or otherwise coupled, therebyconstituting an arrangement of one or more physical units or devices.

According to embodiments, the IR imaging device or arrangement 1 mayfurther comprise a visible light imaging system 11 having a visualsensor 16. The IR imaging device or arrangement 1 further comprises atleast one memory 15.

The capturing of IR images is performed by IR imaging system 12comprised in the IR imaging device or arrangement 1. Optionally, alsovisual images are captured by a visible light imaging system 11comprised in the IR imaging device or arrangement 1. The captured one ormore images are transmitted to a processing unit 2 capable of performingimage processing operations, comprised in the IR imaging device 1. Thecaptured images may also be transmitted with possible intermediatestoring to a processing unit comprised in the IR imaging arrangement 1and being coupled to, but physically separate or external from, theimaging systems 12, 11. According to an embodiment, the processing unitis configured to receive and process infrared image data from the IRimaging system 12. The processing integrated in the IR imaging device orthe separate processing unit coupled to the IR arrangement are providedwith specifically designed programming or program code portions adaptedto control the processing unit to perform the steps and functions ofembodiments of the inventive method described herein.

The processing unit 2 may be a processor such as a general or specialpurpose processing engine for example a microprocessor, microcontrolleror other control logic that comprises sections of code or code portions,stored on a computer readable storage medium, that are fixed to performcertain tasks but also other sections of code, stored on a computerreadable storage medium, that can be altered during use. Such alterablesections can comprise parameters that are to be used as input for thevarious tasks, such as the calibration of the imaging device orarrangement 1, the sample rate or the filter for the spatial filteringof the images, among others.

According to an embodiment, the processor or processing unit 2 isconfigured to process infrared image data from the infrared sensordepicting a scene. According to further embodiment, the processor isconfigured to receive a first IR image depicting the scene and beingcaptured at a first time instance; receive a second IR image depictingthe scene and being captured at a second time instance; perform imageprocessing operations on image data derived from the first and second IRimages to generate a collection of data representing the location of gasin the image; and generate a third image, e.g. being a gas visualizingimage, by adjusting pixel values, in an image depicting the scene,dependent on the collection of data.

According to an embodiment, the processing unit 2 is configured togenerate a third image, or gas visualization image, by adjusting pixelvalues, in an image depicting the scene, dependent on the collection ofdata. According to an embodiment, the processing unit 2 is configured togenerate a third image, or gas visualization image, by adjusting pixelvalues of pixels corresponding to gas representing pixels in the firstor second captured IR image, such that the gas representing pixels aredistinguishable in the gas visualizing image.

According to an embodiment, the processing unit 2 is configured todetect gas within the scene by detecting gas representing pixels in thefirst or second IR image based on the collection of data.

According to one or more embodiments of the present invention, theprocessing unit 2 is configured to perform the steps according to any orall of the method embodiments presented herein. According to anembodiment, the processing unit 2 is configurable using a hardwaredescription language (HDL).

According to an embodiment, the processing unit 2 is aField-programmable gate array (FPGA), i.e. an integrated circuitdesigned to be configured by the customer or designer aftermanufacturing. For this purpose embodiments of the invention compriseconfiguration data configured to control an FPGA to perform the stepsand functions of the method embodiments described herein.

In this document, the terms “computer program product” and“computer-readable storage medium” may be used generally to refer tomedia such as a memory 15 or the storage medium of processing unit 2 oran external storage medium. These and other forms of computer-readablestorage media may be used to provide instructions to processing unit 2for execution. Such instructions, generally referred to as “computerprogram code” (which may be grouped in the form of computer programs orother groupings), when executed, enable the IR camera 1 to performfeatures or functions of embodiments of the current technology. Further,as used herein, “logic” may include hardware, software, firmware, or acombination of thereof.

The processing unit 2 communicates with a memory 15 where parameters arekept ready for use by the processing unit 2, and where the images beingprocessed by the processing unit 2 can be stored if the user desires.The one or more memories 15 may comprise a selection of a hard diskdrive, a floppy disk drive, a magnetic tape drive, an optical diskdrive, a CD or DVD drive (R or RW), or other removable or fixed mediadrive.

Further Embodiments

According to an embodiment, the user can save the final gas image 1100or any of the previous images corresponding to the different methodsteps to the memory 15 for later viewing or for transfer to anotherunit, such as a computer, for further analysis and storage.

In an alternative embodiment, disclosed methods can be implemented by acomputing device such as a PC that may encompass the functions of anFPGA-unit specially adapted for performing the steps of the method forone or more embodiments of the present invention, or encompass a generalprocessing unit 2 according to the description in connection with FIG.6. The computing device may be a part of an IR imaging arrangement 1 andbe communicatively or otherwise coupled to the units and functionalityof the IR imaging arrangement 1. The computing device may furthercomprise the memory 15 and also the display unit 3. It would be possibleto use the disclosed methods live, i.e. for a streamed set of imagesfiltered and combined in real time, or near real time, for instance at30 Hz, that can be recorded and replayed as a movie, but it would alsobe possible to use still pictures.

According to an embodiment the IR camera comprises a visual imagingdevice and the processor 2 is adapted to perform fusion. According tothis embodiment, the image depicting the scene 1200, which is combinedwith an intermediate gas image 900 or an intermediate gas image withadjusted pixel values 1000 to generate a final gas image 1100, may be avisual image may be used as. Alternatively, the detected gas may beadded to and colored in a visual image and an IR image, respectively.

According to an embodiment, the final gas image 1100 comprising avisualization of gas present in the scene is presented to the user ofthe IR camera on a display 3 comprised in, or communicatively coupledto, the IR camera.

According to an embodiment, the processing unit 2 is adapted to performimage processing operations on image data derived from the first andsecond IR images to generate a collection of data representing thelocation of gas in the image.

According to an embodiment, the processing unit 2 is adapted to detectgas within the scene by detecting gas representing pixels in the firstor second IR image of the previous step based on the collection of data.

According to an embodiment, the processing unit 2 is adapted to generatea gas visualizing image, also referred to as a final gas image 1100, byadjusting, in an image 1200 depicting the scene, pixel values of pixelscorresponding to gas representing pixels in the first or second IR imageof the previous step, such that the gas representing pixels aredistinguishable in the gas visualizing image.

According to an embodiment, the IR imaging device or arrangement 1further comprises interaction functionality 4, enabling the operator ofthe IR imaging device or arrangement 1 to provide input to the IRimaging device or arrangement. According to an embodiment, theinteraction functionality comprises a selection of one or more controldevices for inputting commands and/or control signals, e.g. aninteractive display, joystick and/or record/push-buttons.

According to an embodiment, the operator may adjust the sensitivity ofthe gas detection with relation to how much of the gas present in thescene that is to be detected versus how much noise in the image thatwill be wrongfully interpreted as gas and detected by the gas detectionmethod.

According to an embodiment, the sensitivity level value is a function ofthe two thresholds of the optional edge detection steps S520 and S540,and the threshold of the gas detection step 412. Thereby, the adjustmentof the combined sensitivity value will lead to adjustments of allthresholds, according to a predetermined function.

According to another embodiment, one or more of the thresholds for steps414, S520 and S540 may be adjusted separately, or in combination.

An increased threshold value for steps S520 and S540 leads to a lesssensitive detection, possibly allowing more gas information to beinterpreted as edges and thereby removed, while an increase of thethreshold value for step 412 leads to a more sensitive detection whereinpixels with pixel values within a larger range are detected ascomprising gas information.

Thereby, if the operator wants to be sure that no gas information ismissed in the detection, the sensitivity may be manually decreased,while if the operator wants to be sure not to include anything but gasin the detection result, the sensitivity may be manually increased. Theoperator may further adjust the gas sensitivity up and down, viewing theresult on the display 3 of the IR imaging device or arrangement 1, untilthe operator is satisfied with the displayed result.

Use Case

In an exemplifying use case of an embodiment, an operator of an IRimaging device 1 aims the IR imaging device at target scene, whereinpresence of gas, for example gas leaking from a pipe, is suspected.

While aiming the IR imaging device 1 at the target scene, the operatoris presented with an image wherein any gas present in the scene isvisualized, in real time or near real time, on a display 3 of the IRimaging device 1. Any gas present in the scene is detected andvisualized for every image frame in the sequence of image frames that iscaptured by the IR imaging device 1. Since the detected gas isvisualized for every frame, the operator viewing the displayed imagesequence on the display 3 of the IR imaging device 1 will see a movingvisualization of gas, wherein the visualized gas is correctly located inrelation to the rest of the scene in real time. This is enabled byvisualizing gas onto, or approximately onto, the pixels in the imagewhere gas information has been detected.

According to embodiments of the invention the visualized gas has correctsize and distribution, since it is visualized onto the pixels that havebeen identified as comprising gas information.

According to embodiments of the invention the visualized gas has agas-like, or cloud-like, appearance, for instance by the gas informationbeing superimposed onto the image depicting the scene in a non-opaquemanner and/or by the gas information being colored using more than onecolor hues and/or saturation levels. Furthermore, since the gasvisualization is updated for every frame and correctly or at leastnearly correctly depicts the shape of the detected gas, the operatorwill experience that the visualized gas moves continuously in a gas-likemanner as the visualization changes from frame to frame.

Further Advantages

The method according to the inventive embodiments improves the gasdetection and visualization performance of any type of IR camera,whether it is a camera with cooled detectors or with uncooled detectors,since the method makes the detection more accurate due to the fact thatsmaller differences in the IR images will be made distinguishable.

For uncooled detector cameras it is typically hard to distinguish adifference between data of different wavelengths. Furthermore, theimages often comprise a lot of noise. The image processing steps of theinventive method thereby enable gas detection in such images, where itwas not previously possible.

For cooled detector cameras, the gas detection and visualizationperformance is improved compared to previous methods.

As can be readily understood by a person skilled in the art, if the IRcamera used is adapted to a certain wide or narrow band of wavelengthradiation it will still obtain the advantageous gas detection andvisualization capabilities.

According to an embodiment, the IR camera may be a single band IRcamera, meaning that the IR camera is adapted to receiving radiationwithin a certain range of wavelengths and creating images showing avisible representation of said received radiation. This adaptation ofthe IR camera, i.e. fixed wavelength range sensitivity, may beaccomplished either by the use of fixed optical elements between thedepicted scene and the IR sensor 20, or by adjustable physical/hardwareor software optical elements, such as for instance dual band filters orfilter wheels, that are temporarily fixed. To temporarily fixate theoptical elements would result in the IR camera working as a single bandIR camera, i.e. an IR camera with fixed wavelength range sensitivity.

According to another embodiment, the IR camera may be a single band IRcamera, meaning that the IR camera imaging systems are adapted toreceiving radiation within a certain range of wavelengths and creatingimages showing a visible representation of said received radiation.

Some examples of gases that may be of interest to detect and visualizeare: 1-Pentene, Benzene, Butane, Ethane, Ethanol, Ethyl benzene,Ethylene, Heptane, Hexane, Isoprene, MEK, Methane, Methanol, MIBK,Octane, Pentane, Propane, Propylene, Toluene and/or Xylene.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention mayinclude only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description, butis only limited by the scope of the appended claims.

1. A method for gas visualization in an infrared (IR) image depicting ascene, the method comprising: capturing a first IR image depicting thescene at a first time instance and a second IR image depicting the sceneat a second time instance; performing image processing operations onimage data derived from said first IR image and from said second IRimage, to generate a collection of data representing the location of gasin one of the first or second IR image; and generating a third image byadjusting pixel values in an image depicting the scene, dependent onpixel values of said collection of data.